Holistic Video Stitching for Street Panorama

In this paper, we address how to automatically generate a panorama for a street view from a long video sequence. We model the panorama as a low-rank matrix and formulate the problem as one of ro­ bust recovery of the low-rank matrix from highly incomplete, corrupted, deformed measurements (the video frames). We leverage powerful high­ dimensional convex optimization tools from compressive sensing of sparse signals and low-rank matrices to solve this problem. In particular, we show how the new method can effectively remove severe occlusions or corruptions (caused by trees, cars, or reflections etc.), and obtain clean intrinsic street panoramas that are consistent with all frames. We also show how our method can automatically and robustly establish pixelwise accurate registration among all the video frames. We demonstrate the effectiveness of our method by conducting extensive experimental comparison with other popular video stitching methods such as AutoStitch and Adobe Photoshop.